Impact Initiatives: Analytical Process

Statistics Without Borders 2024

Elham Ali

Overview

This presentation provides an review of findings from relevant analytical process documents from Impact Initiatives, best practices, and recommendations for improvement. The documents reviewed in this presentation are:

Guidelines_2-3_Quant Data Analysis_Annex3
Github: cleaningtools
Github: analysistools
Github: presentresults

Executive Summary

TBD

Problem Statement

The Impact Initiatives team have faced challenges to solve for the below pain points. This report aims to solve for these issues and provide best practices in general.

Handle missing data.
Correct size bias in the assessment.
Provide calibration techniques to help with better estimates.

Example

This report uses the 2023 Ukraine | REACH Health Sector Need Assessment dataset to illustrate examples of recommendations shared in this presentation.

Guidelines_2-3_Quant Data Analysis_Annex3

Key sections

[1. “Sampling Frame” Section]
[2. “Sample” Section]
[3. “Extended DAP” Section]

1. “Sampling Frame” Section

About

The “Sampling Frame” section includes information on cluster names, strata variables, and population data. It focuses on different localities, administrative levels, and household numbers.

Best practices observed:

Use of stratified sampling with multiple strata levels (e.g., locality, admin levels).
Clear listing of variables and population numbers for different areas.

Finding
Example
Recommendation

Stratified sampling: In the UKR2217a dataset, there is a variable ‘oblast’ that covers four areas: Vinnytska, Dnipropetrovska, Zaporizka oblasts and Kharkiv city. This is used to demonstrate how stratification works and that these areas are included in the sample. There will be demographic changes due to high-risk conflict zones with forced migration and displacement. This means the frame needs to regularly updated to reflect these changes.

cluster name variable	first strata name variable	population
oblast	Admin1	HH_number
UA12	Dnipropetrovska	[population number]
UA63	Kharkivska	[population number]
UA05	Vinnytska	[population number]
UA23	Zaporizka	[population number]

Ensure the frame is regularly updated to reflect demographic changes. If there’s data indicating new settlements or demographic shifts (like a ‘date of settlement’ column), it could be used to show the need for regular updates in the sampling frame.¹
Population units: Reflect the current population structure post-conflict.
Cluster names: Ensure they match the administrative divisions in the dataset.

In this table:

“cluster name variable” corresponds to the administrative division codes (oblast).
“first strata name variable” should be the name of the administrative division as listed in the dataset.
“population” should reflect the latest population figures for each strata, which might require research or demographic data.

You would replace “[population number]” with the actual population numbers for each administrative division. If you have access to updated population numbers or local sources that can provide recent data, those should be used to fill in the population column.

Recommendation:

The recommendation for regular updates for demographic changes means that the sampling frame should not only include the current administrative divisions but also be updated periodically to reflect any new developments or population movements.

In the table provided, demographic changes would be reflected by adjusting the “population” figures regularly to match the current demographics. If there are new settlements or cities within the oblasts, these should be added as new rows, ensuring that the sampling frame captures the entire population accurately and is up-to-date with any shifts or changes in the population structure.

This dynamic approach maintains the sampling frame, which involves periodic revision and validation with the latest demographic data, rather than using a static set of figures.

Importance

This recommendation is important for several reasons:

Accuracy: Reflects the current demographic situation, ensuring that the sample is truly representative of the population at the time of the survey.
Relevance: Ensures that the data collected is pertinent to the present context, making findings more applicable for decision-making.
Equity: By including new settlements or population groups, the analysis considers potentially marginalized or newly formed communities.
Validity: Reduces sampling bias, increasing the validity of the survey results and subsequent analyses.

Regular updates help ensure that the survey design remains robust and that the insights derived from the data are accurate and relevant to the current situation.

Optional

Validate the frame with local sources or GIS data for accuracy especially if:

The geographic scope of the assessment includes areas with dynamic population distributions or recent changes due to migration, natural disasters, or other factors.
There is a need to ensure that all sub-populations within the geographic coverage are represented accurately, which is critical in humanitarian contexts like in Ukraine.

2. “Sample” Section

About

The “Sample” section almost mirrors the “Sampling Frame” section with a direct connection between the frame and the actual sample drawn.

Best Practices Observed:

Consistency in the use of strata and cluster names between the sampling frame and the sample.
Representation of diverse localities in the sample.

Finding
Example
Rec 1 - 2
Rec 3 - 4

Consistency in strata and cluster names: The regional or demographic categories in the sampling frame match those in the actual sample data.

cluster name variable	first strata name variable	RDS Used	CHW Verified	number of surveys
locality	Admin1	(Y/N)	(Y/N)
city_1	UA12	Y	N	16
city_2	UA63	N	Y	6
city_3	UA05	Y	Y	17

Other sampling techniques: If the dataset has variables indicating transient populations, we could account for respondent-driven sampling.¹
Cross-verification with community workers: If there are variables indicating community health worker interactions, these could be used to cross-check the representativeness of the sample.

Number of surveys: Adjust planned versus actual surveys conducted in each stratum.
Selected units: Document any inaccessible areas and remove them from the sample.

In this table:

“RDS Used” (Respondent-Driven Sampling Used) is a binary indicator (Y/N) that marks whether RDS was used for sampling in a particular locality.
“CHW Verified” (Community Health Worker Verified) is another binary indicator showing whether community health workers have verified the representativeness of the sample for each locality.
The locality and Admin1 columns should match the actual data points in your dataset.

Recommendation

Consistency in Strata and Cluster Names:
- Update the “locality” and “Admin1” columns to match the actual administrative names and codes used in your dataset.
- For example, replace “city_1” with the actual city name and “governorate_1” with the correct administrative region such as “UA12” for “Dnipropetrovska”.
Innovative Sampling Techniques:
- If there are variables indicating areas with high mobility or transient populations, consider implementing respondent-driven sampling techniques. This could involve adding a column to indicate whether respondent-driven sampling was used for particular localities.
Cross-Verification with Community Health Workers:
- If community health workers provided data, include an additional column in your tab to record their input on the representativeness of the sample for each locality.

Adjust the “number of surveys” column to reflect the actual number of surveys conducted in each area.

Importance
Adding these columns provides a nuanced view of the sampling methodology, which is crucial for:

Respondent-Driven Sampling (RDS) Used:
- Identifies non-traditional sampling in areas where standard methods are challenging.
- Improves representativeness in transient or hard-to-reach populations.
Community Health Worker (CHW) Verified:
- Ensures local expertise informs the representativeness of the sample.
- Enhances data reliability, especially in areas where formal data collection is difficult.

These additions aid in understanding and improving sampling strategies, leading to more reliable and actionable insights from the analysis.

3. “Extended DAP” Section

About

The “Extended DAP” (Data Analysis Plan) section includes research questions, hypotheses, indicators, and variables. It guides the analytical approach for specific research questions.

Best Practices Observed:

Structured approach with clear hypotheses, indicators, and variables.
Focus on relevant research questions with a defined hypothesis type.

Finding
Rec 1
Rec 2
Rec 3

Structured approach to data analysis: The dataset has different health indicators, which can be used to demonstrate structured analysis around hypotheses like “Access to clean water is less than 50% in rural areas”.

Primary research question: What is the impact of proximity to conflict zones on health service availability?
Sub-research questions: How does the damage to infrastructure correlate with disruptions in medical services? What are the critical staffing shortages affecting healthcare delivery?
Type of analysis: Define analyses planned versus those conducted.
Hypotheses: Link each analysis to the hypotheses it’s intended to test.

Advanced statistical models: Identify variables that demonstrate hierarchical or complex data structures, such as patient outcomes within hospitals or schools within communities. For example, if the Ukrain dataset includes health outcomes by hospital and each hospital serves a specific community, so use mixed-effects models to account for the nesting of patients within hospitals.¹

Enhanced data visualization: Use the geographic data in the dataset to create maps that display health indicators. For instance, if there are variables for clean water access and location, map these indicators to visually represent how access varies across different areas.

These steps are important because they:

Ensure Rigor: By clearly defining research questions and hypotheses, you ensure that your analysis is rigorous and systematically addresses the objectives of the study.
Provide Depth: Advanced statistical models like mixed-effects models account for the hierarchical nature of data, which can lead to more accurate conclusions about factors affecting health services.
Facilitate Understanding: Data visualization aids in the comprehension of complex information and reveals patterns that might not be apparent from raw data alone, especially when dealing with geographic variations in health service impacts.

More information:

Advanced Statistical Models: If your dataset includes variables that can be nested (like individual responses within hospitals), use mixed-effects models to account for this structure in the data analysis. This will allow you to isolate the effect of the hospital on the response variable, controlling for individual differences.

Github: cleaningtools

About

The cleaningtools package on GitHub provides a comprehensive set of tools for data cleaning and is structured into three main components.

Check: This component includes functions to flag potential issues with data such as personal identifiable information (check_pii), audit durations (create_audit_list, add_duration_from_audit), and outliers (check_outliers).

Best practices observed:

PII Scrubbing: The check_pii function is a good practice to ensure data privacy by identifying potential personal information.
Audit Trails: By checking audit file durations, this tool ensures data quality by confirming reasonable response times.
Outlier Detection: The check_outliers function is critical for identifying and handling anomalies in the dataset which can skew results.

Recommendations for improvement:

Missing data: Implement methods for handling missing data, possibly through imputation techniques or weighted adjustments.
Size bias: Provide tools for correcting size bias, such as scaling factors based on sample versus population size.
Calibration techniques: Include functions for calibration to adjust survey weights based on known population characteristics, which can help in producing better estimates.

To apply these recommendations to the Ukraine dataset, we would:

Assess the extent and pattern of missing data.
Apply correctional weights if size bias is detected in the sample.
Use calibration techniques to align survey results with known benchmarks, improving the accuracy of estimates.

Impact Initiatives: Analytical Process

Overview

Executive Summary

Problem Statement

Example

Guidelines_2-3_Quant Data Analysis_Annex3

Key sections

1. “Sampling Frame” Section

Optional

2. “Sample” Section

3. “Extended DAP” Section

Github: cleaningtools

Github: analysistools

Github: presentresults